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Abstract 

We perform a comparison of the performance and efficiency of four different function eval- 
uation methods: black-box functions, binary trees, n-ary trees and string parsing. The test 
consists in evaluating 8 different functions of two variables x, y over 5000 floating point values 
of the pair (x,y). The outcome of the test indicates that the n-ary tree representation of 
algebraic expressions is the fastest method, closely followed by black-box function method, 
then by binary trees and lastly by string parsing. 
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Important warning. There is a mistake in the test code that invalidates the most important 
result of this paper, i.e. that n-ary tree based function evaluation is faster than black-box function 
evaluation. It is not true: it is slower by about an order of magnitude. However it is true that 
n-ary tree based evaluation is faster than the other methods discussed in this paper. 

1 Introduction 

In this article we describe a test designed to measure the comparative efficiency of four different 
function evaluation methods (see section |]for details): 

• black-box functions; 

• binary tree representation; 

• n-ary tree representation; 

• string parsing. 



Because of the huge number of parameters involved in such a test (efficiency of compiler, quality 
of test source code, type of hardware, type of test functions, number of variables, object code 
optimization level, and so on) it is evident that this test can be neither definitive nor undebatable. 
However, the results indicate a clear winner in the n-ary tree representation, closely followed by 
black-box functions and binary tree representation. Last (expectedly) comes string parsing. It is 
somewhat of a surprise to discover that n-ary tree representation gathers better results than the 
precompiled black-box functions method. This finding is discussed below (section ^). 
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Existing literature in this topic is scarce or non-existent. Some of the early work focused 
on how to handle floating point computation efficiently, rather than on the actual method used 
for the evaluatio n ]Ash64 |; in other instances, the evaluation of certain classes of functions (e.g. 
polynomials, see [Fik67Q was investigated. 



Most people tend to use the black-box functions method by exploiting the compiler's capabil- 
ities in this sense. Further efforts for better evaluation methods are usually sought after only in 
connection to very specific functions and problems, e.g. in astronomy [SchOC], in number theory 



[ Odl90 ], when using discrete/boolean functions [ MMS + 95 ] 



A novel evaluation method, based on threaded binary trees, was proposed in [ KP97 |; this 
method partially eliminates the cost of recursion by "threading" the binary expression tree before 
the evaluation. Because the operation of threading the binary tree is recursive in nature, the CPU 
time savings are only possible if the same tree is evaluated many times (as is the case in most 
engineering applications). However, the benefits of this approach should decrease with the use of 
n-ary trees, as threading a list of like operators in an n-ary tree has no effect. 

The test consists in evaluating eight different functions of one and two variables x, y over 5000 
randomly generated pairs of values for (x,y), all in the interval [0, 1]. The functions are: 



1. x; 

2. x + y; 

3. x y ; 

4. (x + y)x y ; 

5. sin(x); 

6. sin((x + y)x v ); 

7. x + y + l; 

8. 2xy{x + y + 1). 



The above functions have been chosen as a representative set for unary and binary operators, in 
the sense that both binary operators (sum, power) and unary operators (the sine function) are 
present. Little does it matter for the outcome of the test that not all operator types have been 
employed, for the time taken to carry out floating point computation would have been exactly the 
same in all cases. 

The number of variables has been limited to two for simplicity. However, for reasons which will 
become apparent below (see section [2.3| about the description of the n-ary tree representation), 
adding more variables to the expressions would only have served to overemphasize the outcome of 
the tests, especially in the case where long sequences of like operands are employed (e.g. linear 
equations, or products of the kind X1X2X3 ■ ■ ■ x n ). 

The test code has been written in pure C in order to minimize the effect of compiler overhead, 
and compiled with the GNU C Compiler version 2.95.2. All optimization options have been tested 
(-0, -02, -03) as well as "debug" mode and "normal" (no flags) mode. In all cases the test results 
have been consistent with the order: n-ary trees, black-box functions, binary trees, string parsing. 

The test has been run on a Pentium-Ill 450MHz with 192MB RAM with the Linux operating 
system. All results have been obtained by running the executables in single-user mode and by 
flushing RAM caches after each run. Enabling the caches and running in multi-user modes gathers 
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similar results but occasionally the black-box functions wins out (by very little indeed) over the 
n-ary tree method. However, situations where the black-box functions method wins over the other 
methods cannot be replicated; they depend very much on the behaviour of the caching code and 
on the general load of the machine at any given moment. In any case, this kind of outcome only 
occurs when repeatedly runnning the same executable over and over again as different processes, 
not when calling the same function many times within the same process. 

Section ^ describes the four types of evaluation methods tested. Section || describes the imple- 
mentation of the methods. Section || discusses the results of the test. The code used to run the 
test can be downloaded, inspected and reused under the GNU public license from 



tittp : //liberti . dhs . org/liberti/ evaltest . 

2 Evaluation Methods 

In this section we shall carry out a theoretical analysis of the evaluation methods tested in this 
article. 

2.1 Black-box Functions 

This method for function evaluation is by far the easiest to implement and the most commonly 
used within the scientific community, especially where test code has to be rigged up or once-only 
jobs need to be run. It basically lets the compiler do the work of parsing the expression into a 
binary tree which is hardwired in the object code at compile time. Evaluations are supposed to 
be very fast (mainly because most of the work is carried out once only at compile time); its main 
drawback is that changes to the function formulation entail recompilation of the source code, which 
for most pieces of software is not an acceptable solution. 

The programming paradigm for black-box functions follows the lines of the pseudocode below: 

main: 

float x, y, f; 

f = blackbox(x, y) ; 
end 

function blackbox (float x, float y) : 
float z; 

z = sin((x + y)*x~y); 
return z; 
end function 

The reason why this method is called "black-box functions" is that from the main procedure 
point of view, the function is really a black box, in the sense that apart from knowing what 
argument it requires, there is no run-time control over it. 

2.2 Binary Trees 

This representation is based on the idea that operators, variables and constants are nodes of a 
digraph; binary operators have two outcoming edges and unary operators have only one; leaf nodes 
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have no outcoming edges (for graph-related terminology and definitions, see [ Har7l| , [ KVO0[ ) . For 
example, the function x + y + 1 would be represented as in fig. [y. Where unary operators are 




Figure 1: Binary tree representation for x + y + 1. 



employed, a dummy second operand is often used. 

This type of function representation is the most commonly used where there is a need for some 
degree of run-time control over the definition of the mathematical function being represented. 
However, it should be noted that performing algebraic operations on this representation is not 
overly simple. Most software that does not do symbolic manipulation of algebraic expressions 
employs this kind of representation. Furthermore, most general-purpose compilers (including the 
GNU C compiler) use this representation too. 



2.3 n-ary Trees 

This technique is a combination of binary trees and lists. Operators can have any number of 
operands. This allows for much more efficient handling of sequences of like operands, e.g. in linear 
expressions or long products {x\X2 ■ ■ ■ x n ). For example, the function x + y+l would be represented 
as in fig. g. 




Figure 2: n-ary tree representation for x + y + l. 



This type of function representation is often employed when symbolic manipulation is required. 
The data structures used by languages like Prolog and especially LISP are very similar in concept 
to n-aiy trees. 
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2.4 String Parsing 

String parsing is a process by which a string containing an algebraic expression is evaluated directly 
without middle steps like tree representation. String parsers usually consist of a lexical analyser, 
which returns tokens (operators), symbols and constants, and a grammatical interpreter which 
drives the lexical analyser on the given string. It then performs the mathematical operations 
signalled by the tokens on the operands. When a symbol is returned, it is looked up on a symbol 
table to discover its numeric value. Because of the design complexity, it is to be expected that 
string parsing is slower than other methods. It is mainly employed where the parsing is to be 
carried out once only (possibly as a pre-processing step to some main algorithm). 

By changing the grammatical interpreter, a string parser can be used to build binary or n-ary 
trees for algebraic expressions input as strings. This is how compilers transform source code (i.e. 
strings) into object code. 



3 Implementation in the C Language 

The implementation of the techniques described in section |^ above has been carried out in C in 
order to minimize the amount of compiler-generated overhead code, as the C language has very 
low requirements in this respect. Furthermore, no external library has been used as it would have 
invalidated the timing tests. Instead, all the code necessary to the test has been written from 
scratch. 

The part of the test concerning black-box functions was the easiest to code. No particular 
coding technique was employed. Functions returning results of the test expressions were compiled 
into the executable and called from the main routine. 

Tree handling, in both the binary and n-ary forms, required more work. A tree, for the purposes 
of this test, is defined as follows. 



struct tree { 
int optype ; 
long var index; 
double value ; 
struct tree** nodes; 
long nodesize; 

}; 



// operator type 

// variable index if node is a variable 
// value if node is a constant 
// subnodes if node is not a leaf node 
// number of subnodes 



The above definition is generic enough to be able to accommodate both binary and n-ary trees. 
For binary trees, nodesize is always set to 2. No "string to tree" parser has been included as the 
test code did not need that kind of generality; all the function trees (both binary and n-ary) have 
been manually coded in. 



The string parser has been derived from the ideas given in ftr9§. The parser code given 



therein has been modified to support exponentiation and unary functions in the form f(x) 



3.1 Code Validation 



The validity of the code has been verified along with the test proper. All results from evaluations 
with the four methods described above coincide (up to at least three significant digits) for each of 
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Test Results 


Compiler Code Optimization Level 


Evaluation Method 


Normal 


Debug (-g) 


-0 


-02 


-03 


Black-box functions 


0.76s 


0.76s 


0.75s 


0.74s 


0.73s 


Binary trees 


0.84s 


0.84s 


0.80s 


0.82s 


0.82s 


n-ary trees 


0.73s 


0.74s 


0.70s 


0.70s 


0.70s 


String parsing 


2.88s 


2.97s 


2.63s 


2.23s 


2.56s 



Table 1: Test results. Values are expressed in seconds of "user" CPU time (time spent on system 
calls was 0.00s in all cases). 

the test expressions. 



4 Results 

As has been mentioned in the introduction, the test consists in evaluating the eight expressions 
above (see page|2|) over 5000 randomly generated pair values for (x, y) (all in the interval [0, 1]) with 
each of the four described evaluation methods, trying all the possible compiler code optimization 
flags. The test is carried out in a single-tasking environment where memory cache has little or no 
effect. The results of the test are reported in table [jj 

These results are surprising because normally we would expect black-box functions to be the 
most efficient evaluation method, whereas the actual "test winner" is the n-ary tree representation 
(although, as has been noted in the introduction, this test is far from definitive — the parameters 
that can affect performance are too many to be controlled all at once) . This result is strengthened 
by the consideration that black-box functions are so easy to program that the test cannot be 
invalidated because of "programming errors" or inefficient coding. On the other hand, it may be 
true that with more careful coding, the results referring to tree evaluation could be made even 
better. 

4.1 n-ary Trees: the Best Evaluation Method 

In order to explain this result, one has to consider the similarities between black-box functions 
and binary trees. Although from the programmer's point of view the two methods are far from 
similar, the resulting machine code need not be all that different. As has been explained earlier, 
most compilers work in such a way as to hard-code binary trees representing the expressions within 
the object code. This binary tree structure may be hidden behind a simpler logic flow than that 
generated by the binary tree method, but the operations are carried out much in the same order 
in both methods. Thus, it comes to no surprise that the binary tree method gathers results which 
are worse, but not by much, than those of black box functions. The two methods are very similar, 
but in the black-box function case the code is created directly by the compiler and can be better 
optimized. 

The n-ary tree method performs in the same way as the binary tree method, except where 
sequences of like operands with length greater than 2 appear in the expressions (e.g. in linear 
expressions with at least 3 nonzero coefficients), where it performs much faster. The evaluation 
algorithm is as follows: 



double eval (struct tree* expression, double varvalues) { 
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int i ; 
double ret ; 

switch(expression->optype) { 
case CONSTANT: 

return expression->value ; // leaf node 

case VARIABLE: 

return varvalues [expression->varindex] ; // leaf node 

case SUM: 
ret = 0; 

for (i = 0; i < expression->nodesize ; i++) { 

ret += eval(expression->nodes [i] , varvalues); // recursive call 

} 

return ret ; 
case PRODUCT: 
ret = 1; 

for (i = 0; i < expression->nodesize ; i++) { 

ret *= eval(expression->nodes [i] , varvalues); // recursive call 

} 

return ret ; 
// ... all other cases 
} 

} 

Consider the cases depicted in fig. [j] and[2| In the first case, where the nodesize is always 2, 
a recursive function call has to be performed for each of the two '+' operators; in the second case, 
however, only one recursive function call needs to be carried out (for the only '+' operator). 

This explains the advantage of using n-ary trees in evaluation of algebraic expressions. The 
computational cost of generating a recursive function call is high for most compilers. It becomes 
evident that the longer "like operand sequences" are, the better this evaluation method becomes. 

4.2 Results on String Parsing 

String parsing, although definitely the worst method, was also somewhat surprising in that it 
was not as bad as a superficial analysis would suggest. After all the code complexity of a lexical 
analyser and a grammatical interpreter is far greater than the other evaluation algorithms presented 
above. However, especially when memory caching was allowed, the timings of the string parsing 
method got better and better. The best result we obtained was close to 1.00s; so even though 
it still worse than the other methods, it was the technique that benefited most from caching (in 
different processes, however, not in the same process). However, the same test carried out using 
the popular Unix utility be (in most implementations based around the lex and yacc compiler 
tools) gathered an appalling result of over 26s of user CPU time, notwithstanding the fact that 
Stroustrup's adapted parser is highly recursive in nature whereas lex and yacc usually generate 
non-recursive (and hence theoretically faster) code. 



5 Conclusion 

In this paper we have analysed the performance of four common evaluation methods: black-box 
functions, binary trees, n-ary trees and string parsing. The result of the test indicates that the 
n-ary tree expression representation is the best function evaluation method. 
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